Likelihood ratio tests in linear mixed models with one variance component

We consider the problem of testing null hypotheses that include restrictions on the variance component in a linear mixed model with one variance component. We derive the finite sample and asymptotic distribution of the likelihood ratio test (LRT) and the restricted likelihood ratio test (RLRT). The spectral representations of the LRT and RLRT statistics are used as the basis of an efficient simulation algorithm of these null distributions. The large sample chi-square mixture approximations using the usual asymptotic theory for a null hypothesis on the boundary of the parameter space (e.g., Self and Liang 1987, 1995), has been shown to be poor in simulation studies. Our asymptotic calculations explain these empirical results. The theorems of Self and Liang are perfectly correct, but we show that their assumptions do not, in general, hold for linear mixed models. Moreover, the hope that these assumptions could be weakened so that the Self and Liang theory would apply to linear mixed models proved false, since these asymptotic distributions do not hold for linear mixed models. One-way ANOVA and penalized splines models illustrate the results.